Information processing device, robot manipulating system and robot manipulating method

ABSTRACT

A robot manipulating system includes a game terminal having a game computer, a game controller, and a display configured to display a virtual space, a robot configured to perform a work in a real space based on robot control data, and an information processing device configured to mediate between the game terminal and the robot. The information processing device supplies game data associated with a content of work to the game terminal, acquires game manipulation data including a history of an input of manipulation accepted by the game controller while a game program to which the game data is reflected is executed, converts the game manipulation data into the robot control data based on a given conversion rule, and supplies the robot control data to the robot.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part of U.S. patent applicationSer. No. 17/267,447 filed Feb. 9, 2021, which is the U.S. National Stageof International Application No. PCT/JP2019/031479 filed Aug. 8, 2019,which claims benefit of priority to Japanese Patent Application No.2018-151917 filed Aug. 10, 2018, and Japanese Patent Application No.2019-105728 filed Jun. 5, 2019, the entire contents of which areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a technology which unites a computergame and an industrial robot.

BACKGROUND ART

Conventionally, a technology which combines a computer game and a robotwhich is an actual object is proposed. Patent Documents 1 and 2 disclosethis kind of technology. Note that the “computer game” as used herein isa general term for a game which operates on a computer and is performedbetween the computer and an actual human being, and it is also referredto as a “video game” etc.

Patent Document 1 discloses that an industrial robot for welding andlaser machining is provided with a game controller for a home computergame terminal as a controller for teaching and manipulational inputting.Correlation between the game controller to the industrial robot isperformed through a well-known program using a personal computer whichis commercially available.

Patent Document 2 discloses a system provided with a real space where aplurality of physical agents (for example, gaming robots such asvehicles and their accessories) exist, a controller which receives anoperating command to the physical agent by a user input, and a hostdevice which mediates between the physical agent and the controller. Thesystem performs a game which causes the plurality of physical agents tocompete in the real space (for example, a car racing game). A hostdevice operates the physical agent so that a virtual space under theuser's control and the real space maintain a real-time equivalence tochange a state of the real space or change a state of the virtual space.

REFERENCE DOCUMENTS OF CONVENTIONAL ART Patent Documents

[Patent Document 1] JP2012-139746A

[Patent Document 2] JP2015-533534A

DESCRIPTION OF THE DISCLOSURE Problem(s) to be Solved by the Disclosure

A skillful game user can instantly grasp the state of the virtual spaceprojected on a display of a computer-game terminal and input acorresponding command by operating the game controller intuitively andcorrectly. The present inventors have examined utilizing such gameuser's excellent skills as a resource for the industry.

Patent Document 1 describes an effect that the robot operator canmanipulate the industrial robot like a game. However, since the robotoperator actually sees the real space where the industrial robotperforms a work, he/she more strongly feels like working rather thangaming.

In Patent Document 2, since the virtual space under the game user'scontrol has the equivalence to the real space, the virtual spacerecognized by the game user never exceed the real space.

Originally, the game aims at amusement. For the game user, the game inwhich he/she feels like working lacks in interest, which may lead tolowering of his/her motivation. The present disclosure proposes a systemin which a robot performs a work (labor) by a game user inputting amanipulation while playing a game, without spoiling the aspect ofamusement in which the user can enjoy an unusual virtual space.

SUMMARY OF THE DISCLOSURE

An information processing device according to one aspect of the presentdisclosure mediates between a game terminal having a game computerconfigured to execute a game program, a game controller configured toaccept an input of manipulation into the game computer, and a displayconfigured to display a virtual space outputted from the game computer,and a robot configured to perform a work in a real space based on robotcontrol data. The information processing device includes a game datafeeding part configured to supply game data associated with the contentof work to the game terminal, a game manipulation data acquiring partconfigured to acquire game manipulation data including a history of theinput of the manipulation accepted by the game controller while the gameprogram to which the game data is reflected is executed, a firstconverting part configured to convert the game manipulation data intothe robot control data based on a given conversion rule, and a robotcontrol data feeding part configured to supply the robot control data tothe robot.

A robot manipulating system according to another aspect of the presentdisclosure includes a game terminal having a game computer configured toexecute a game program, a game controller configured to receive an inputof manipulation into the game computer, and a display configured todisplay a virtual space outputted from the game computer, a robotconfigured to perform a work in a real space based on robot controldata, and the information processing device described above configuredto mediate between the game terminal and the robot.

A method of manipulating a robot according to another aspect of thepresent disclosure is a method of manipulating the robot configured toperform a work in a real space based on robot control data by using agame terminal having a game computer configured to execute a gameprogram, a game controller configured to accept an input of manipulationinto the game computer, and a display configured to display a virtualspace outputted from the game computer. The method comprising the stepsof supplying game data associated with the content of work to the gameterminal, acquiring game manipulation data including a history of theinput of the manipulation accepted by the game controller while the gameprogram to which the game data is reflected is executed, converting thegame manipulation data into the robot control data based on a givenconversion rule, and supplying the robot control data to the robot.

In the information processing device, the robot manipulating system, andthe robot manipulating method which are described above, since the gamemanipulation data is converted into the robot control data based on theconversion rule, the contents of the computing game which the game userenjoys (i.e., characters, items, effects which appear in the virtualspace of the game, the rule of the game, etc.) and the content of workperformed by the robot do not need to be directly related to each other.That is, the virtual space projected on the display of the game terminalcan be turned into the world unique to the game which is greatlydifferent from the work performed by the robot in the actual world.Therefore, the aspect of amusement of the game in which the unusualvirtual space is appreciated is not spoiled.

Therefore, according to the information processing device, the robotmanipulating system, and the robot manipulating method, the game userwho is also the robot operator can make the robot work by themanipulation inputted during the game play, while purely enjoying thevirtual world of the game.

Effect of the Disclosure

According to the present disclosure, a system is proposed, in which arobot performs a work by a game user inputting a manipulation whileplaying a game, without spoiling the aspect of amusement in which theuser can enjoy an unusual virtual space.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating the entire configuration of arobot manipulating system according to one embodiment of the presentdisclosure.

FIG. 2 is a block diagram illustrating a configuration of a gameterminal.

FIG. 3 is an appearance view of one example of a game controller.

FIG. 4 is a functional block diagram of a mediation server.

FIG. 5 is a block diagram illustrating a configuration of a computerwhich functions as the mediation server.

FIG. 6 is a view illustrating a flow of machine learning by a learningdevice.

FIG. 7 is a view illustrating a function of a game data generatingdevice.

FIG. 8 is a view illustrating a function of a robot control datagenerating device.

FIG. 9 is one example of a game screen displayed on a display of thegame terminal in Application Example 1.

FIG. 10 is a view illustrating one example of work data in ApplicationExample 1.

FIG. 11 is a flowchart of processing by a robot manipulating system.

FIG. 12 is a functional block diagram of a mediation server according toModification 1.

FIG. 13 is a block diagram illustrating Application Example 2 of a robotmanipulating system, which illustrates the entire configuration of therobot manipulating system.

FIG. 14 is one example of the game screen displayed on the display ofthe game terminal in Application Example 2.

FIG. 15 is a view illustrating one example of work data in ApplicationExample 2.

MODES FOR CARRYING OUT THE DISCLOSURE

Next, one embodiment of the present disclosure is described withreference to the drawings.

FIG. 1 is a block diagram illustrating a configuration of a robotmanipulating system 1 according to one embodiment of the presentdisclosure. The robot manipulating system 1 includes a game terminal 3,a mediation server 2, a robot 5, and a communication apparatus 4. Thecommunication apparatus 4 connects the game terminal 3 with themediation server 2 so as to be communicatable to each other, andconnects the mediation server 2 with the robot 5 so as to becommunicatable to each other. The communication apparatus 4 may be acommunication network, such as LAN, WAN, and the Internet. In this robotmanipulating system 1, the game user is a game player who enjoys acomputer game using the game terminal 3 and is a robot operator whomanipulates the robot 5.

Game Terminal 3

The game terminal 3 includes a game computer 31, a game controller 38, adisplay 35, and a speaker 37. As the game terminal 3, a business-usegaming machine, a home video game terminal which uses a television setas a display, a portable crystalline-liquid game terminal, aninformation kiosk terminal, a smartphone, a tablet computer, a desktopcomputer, a laptop computer, etc. may be used. Here, one example inwhich the home video game terminal is adopted as the game terminal 3 isdescribed.

FIG. 2 is a block diagram illustrating an outline configuration of thegame terminal 3. As illustrated in FIG. 2 , the game computer 31 of thegame terminal 3 includes a main processor 311, a sound processor 312, anI/O processor 313, a graphics processor 315, and a memory 316. The soundprocessor 312 and the memory 316 are connected with the I/O processor313 through an internal bus. The main processor 311 and the I/Oprocessor 313 are connected to each other. The main processor 311 andthe graphics processor 315 are connected to each other. The I/Oprocessor 313 is connected with an I/O device, such as the gamecontroller 38, through a serial interface (not illustrated). The I/Oprocessor 313 communicates with an external device, such as themediation server 2, through a communication interface (not illustrated).

The game terminal 3 can download a game program from the mediationserver 2 by communicating with the mediation server 2. The game programand save data which are downloaded are stored in the memory 316. Thegame program and the save data may be stored in a storage 314.

The I/O processor 313 supplies to the main processor 311 a manipulationsignal from the game controller 38 operated by the game user. The mainprocessor 311 reads and executes the game program, and performs a givenarithmetic operation. The graphics processor 315 performs a drawing(rendering) processing according to an instruction from the mainprocessor 311 to generate frame data, and outputs a video signal to thedisplay 35. The sound processor 312 decodes voice data according to aninstruction from the main processor 311, and replays the data to outputit to the speaker 37.

The main processor 311 creates game manipulation data based on themanipulation signal etc. inputted by the game user through the gamecontroller 38, and passes it to the I/O processor 313. The I/O processor313 transmits the game manipulation data to the mediation server 2. Thegame manipulation data includes at least identification information onthe game program, game user's identification information, and a historyof the manipulation inputted by the game user through the gamecontroller 38 while playing the game.

The game controller 38 includes a manipulandum 381, a manipulationdetector 382 which detects an operation amount of the manipulandum 381,and a communication device 383 which outputs the operation amountdetected by the manipulation detector 382 to game computer 31 as amanipulation signal. For example, the manipulandum 381 may be one ofknown manipulanda, such as a touch display, a keyboard, a mouse, a crosskey, a stick, a button, a ball, and a lever, or a combination of these.

FIG. 3 is an appearance view of one example of the game controller 38.The game controller 38 illustrated in FIG. 3 has a contour of asubstantially U-shape. The game user grasps and operates handles 101Land 101R at both wing parts with both left and right hands. Operationbutton groups 110 and 120 and analog sticks 112 and 122 are provided inleft and right upper surfaces of the game controller 38, and an R button111 is provided in a front right surface of the game controller 38, an Lbutton 121 is provided in a front left surface. The operation buttongroup 110 and the analog stick 112 are operated with the game user'sright thumb, and the operation button group 120 and the analog stick 122are operated with the game user's left thumb. Moreover, the R button 111and the L button 121 are operated with the game user's right indexfinger and left-hand index finger, respectively. The game controller 38is further provided with a touch display 130 between the operationbutton group 110 and the operation button group 120.

Robot 5

The robot 5 is an industrial robot which is installed at a field, suchas a factory, and performs a work in a real space, for example. Such anindustrial robot includes a vertical articulated robot, a horizontalarticulated robot, a parallel link robot, a polar coordinates robot, acylindrical coordinates robot, and a rectangular coordinates robot. Notethat, although a single robot 5 is illustrated in FIG. 1 , the robotmanipulating system 1 may be provided with a plurality of robots 5.

In this embodiment, a painting robot which performs a painting work isused as one example of the robot 5. The robot 5 includes a manipulator204, an end effector 206 attached to a tip end of the manipulator 204,and a robot controller 201. The robot 5 further includes a robotmanipulator 213 which accepts an input of manipulation to the robot 5(in detail, the manipulator 204 and the end effector 206). The robot 5may further include a robot vision (not illustrated). The robotcontroller 201 is communicatably connected with the manipulator 204, theend effector 206, and the robot manipulator 213. The robot controller201 is provided with a computer, and by performing decoding andarithmetic processing of the program stored in the processor or varioussignals inputted from the robot manipulator 213 to govern a motioncontrol of the manipulator 204, a motion control of the end effector206, a signal output from various output ports, etc.

In this embodiment, the end effector 206 is a painting gun 206A. Thepainting gun 206A includes a nozzle 207, a paint feeder 202 whichsupplies paint to the nozzle 207, a compressor 205 which pumpscompressed air to the nozzle 207, and a valve device 203 which adjusts adischarge amount of the paint from the nozzle 207 (including ON/OFF ofthe discharge). Operation of the valve device 203 is controlled by therobot controller 201.

A jig 210 which holds a workpiece 211 is disposed near the robot 5.Moreover, near the robot 5, an imaging device CA, whichthree-dimensionally images a workspace of the robot 5 including the jig210, the workpiece 211, and the robot 5, is provided. The imaging deviceCA may image the workspace two-dimensionally. Imaging data generated bythe imaging device CA is transmitted to the mediation server 2 throughthe communication apparatus 4.

Mediation Server 2

FIG. 4 is a block diagram illustrating a configuration of the mediationserver 2. As illustrated in FIG. 4 , the mediation server 2 includes afirst learning device 41, a game data generating device 42, a robotcontrol data generating device 43, and a storage 24. The first learningdevice 41 creates a first learned model using A.I. (ArtificialIntelligence) technology. The game data generating device 42 generatesgame data based on the work data, and provides it to the game terminal3. The robot control data generating device 43 generates robot controldata using the game manipulation data acquired from the game terminal 3,and provides it to the robot 5. The storage 24 stores data used by themediation server 2, such as learned model, learned data, teacher data,game program, and game parameter.

The mediation server 2 includes a calculation controller 20 a and thestorage 24. Each functional part of the mediation server 2 may becomprised of at least one calculation controller 20 a, or two or more ofa plurality of functional parts may be comprised of one calculationcontroller 20 a. As illustrated in FIG. 5 , each calculation controller20 a of the mediation server 2 includes a processor 21, a memory 22 suchas a ROM and a RAM, and an I/O part 25. The storage 24 is connected withthe calculation controller 20 a through an interface 20 b. Thecalculation controller 20 a may be provided with a sole processor 21which performs a centralized control, or may be provided with aplurality of processors 21 which perform a distributed control. Forexample, the calculation controller 20 a may be comprised of at leastone of a computer, a personal computer, a microcontroller, amicroprocessor, a PLD (programmable logic device) such as an FPGA(field-programmable gate array), a PLC (programmable logic controller),and a logic circuit, or a combination of these. The memory 22 or thestorage 24 stores a basic program, a software program, etc. executed bythe processor 21. By the processor 21 reading and executing the program,the calculation controller 20 a achieves a function configured in thesoftware program.

The first learning device 41 includes functional parts of a learned dataacquiring part 41 a, a pretreatment part 41 b, and a first learning part41 c. Learned data DB which stores learned data, teacher data DB whichstores teacher data, and parameter DB which stores game parameters, areconfigured in the storage 24.

FIG. 6 is a view illustrating a flow of machine learning by the learningdevice (first learning device 41). As illustrated in FIG. 6 , thelearned data acquiring part 41 a acquires data to be used by the machinelearning, such as the learned data. The pretreatment part 41 b pretreatsthe learned data to create the teacher data. The pretreatment includesat least one of various processings such as a conversion of data format,an examination of abnormalities, an extraction of data, and a change ina variable identifier and a filename.

The first learning part 41 c learns a correlation between input data andoutput data by the machine learning. In this embodiment, the input datais game manipulation data and the output data is robot control data. Thefirst learning part 41 c performs supervised learning as one example ofa learning algorithm of the machine learning. Generally, the supervisedlearning is an approach of learning a correlativity model for estimatingnecessary output data for new input data by giving in advance a largequantity of known data set (referred to as “teacher data”) of the inputdata and the output data corresponding to the input data, anddiscriminating a feature which suggests the correlativity between theinput data and the output data based on the teacher data. Although thecorrelativity between the input data and the output data issubstantially unknown when the learning algorithm is started, the firstlearning part 41 c identifies the feature gradually as the learningprogresses to interpret the correlativity. When the correlativitybetween the input data and the output data is interpreted to a levelreliable to some extent, the learned result which is repeatedlyoutputted from the first learning part 41 c can be used for estimatingwhat kind of data the output data should become against the input data.That is, the first learning part 41 c can gradually bring thecorrelativity between the input data and the output data close to anoptimal solution with the progress of the learning algorithm.

In this embodiment, the first learning part 41 c inputs the teacher datainto a neural network and learns the relation between the input data andthe output data. Various parameters set to the neural network are storedin the parameter DB. For example, the parameter DB stores the weightetc. of synapses in the learned neural network. The neural network whereeach parameter to be stored is set becomes the learned model.

Returning to FIG. 4 , the game data generating device 42 includes a workdata acquiring part 42 a, a converting part 42 b, and a game datafeeding part 42 c.

FIG. 7 is a view illustrating a function of the game data generatingdevice 42. As illustrated in FIG. 7 , the work data acquiring part 42 aacquires work data from the robot 5 and the imaging device CA. The workdata includes information on the content of a work, and information onthe workpiece 211 which is a work target object. The information on thecontent of work may include the type of work such as machining,conveying, and painting, the type of the end effector 206 to be used forthe work, and parts to be used for the work, such as a screw and paint,for example. The information on the workpiece 211 may include the type,shape, etc. of the workpiece 211, for example.

The converting part 42 b generates game data by converting the work databy using a given conversion rule. The conversion rule is stored in thestorage 24 in advance. The game data may include a game program to beexecuted at the game terminal 3. Moreover, when the game program isinstalled in the game terminal 3 in advance, the game data may includegame parameters applied to the game program. The game data feeding part42 c supplies the generated game data to the game terminal 3.

Returning to FIG. 4 , the robot control data generating device 43includes a game manipulation data acquiring part 43 a, a converting part43 b, and a robot control data feeding part 43 c which outputs the robotcontrol data.

FIG. 8 is a view illustrating a function of the robot control datagenerating device 43. As illustrated in FIG. 8 , the game manipulationdata acquiring part 43 a acquires the game manipulation data from thegame terminal 3. The converting part 43 b generates the robot controldata by converting the game manipulation data by using a givenconversion rule. The robot control data may include position commandinformation on the end effector 206 and operating command information onthe end effector 206. The robot control data feeding part 43 c suppliesthe generated robot control data to the robot 5. The robot 5 performs awork based on the acquired robot control data.

APPLICATION EXAMPLE 1

Below, Application Example 1 in which the robot manipulating system 1having the above configuration is applied to the robot 5 which performsa painting work is described.

Referring to FIG. 1 , the robot 5 which performs the painting work isprovided with the nozzle 207 of the painting gun 206A at a tip end of ahand. The robot manipulator 213 has a first lever A, a second lever B,and a button C. According to the operation inputted into the first leverA and the second lever B, the manipulator 204 operates to change theposition of the nozzle 207.

The first lever A can accept an input of a command of a movement of thenozzle 207 in the left-and-right direction and the front-and-reardirection. A manipulation input signal value inputted by the first leverA changes continuously from 0 to 1 according to the operating positionof the first lever A. The second lever B can accept an input of acommand of a movement of the nozzle 207 in the up-and-down direction. Amanipulation input signal value inputted by the second lever B changescontinuously from 0 to 1 according to the operating position of thesecond lever B. The button C can accept an input of a command of adischarging amount of the paint. The manipulation input signal valueinputted by the button C changes continuously from 0 to 1 according tothe operating position of the button C.

Although the workpiece 211 which is the work target of the robot 5having the above configuration is held by the jig 210, the posture andthe shape are different for different workpieces 211. Since there isirregularity in the surface of the workpiece 211, the robot operator isrequired for high skill in order to paint this surface uniformly.

The game terminal 3 executes a game program of a drop game. FIG. 9illustrates one example of a game screen 92 displayed on the display 35of the game terminal 3. Not the real space where the robot 5 exists buta virtual space peculiar to the game is displayed on the game screen 92.In the game screen 92 illustrated in FIG. 9 , 20 rows of lines 92 a aredisplayed in a center part, and block(s) falls from the topmost part ofeach row by a given drop pattern. A shooting gun 92 b as the game user'soperation target item and a score 92 c are displayed in a lower part ofthe game screen 92. The game user can operate the game controller 38 toinput into the game computer 31 a command related to a shootingdirection and a shooting timing of a bullet of the shooting gun 92 b.

The game controller 38 is provided with a dial X and a button Y as themanipulanda 381. A manipulation input signal value of the dial X changescontinuously from 0 to 1 with a turning angle of the dial X. The dial Xcan move the shooting gun 92 b in the left-and-right direction of thegame screen 92. The manipulation input signal value of the dial Xcorresponds to the shooting direction of the shooting gun 92 b. Theshooting direction of the shooting gun 92 b is oriented toward theleftmost end of the screen when the manipulation input signal value ofthe dial X is 0, is oriented toward the rightmost end of the screen whenthe manipulation input signal value is 1, and is oriented toward thecenter of the screen in the left-and-right direction when themanipulation input signal value is 0.5. The manipulation input signalvalue of the button Y becomes a value of 0 or 1 depending on whether thebutton Y is pushed. The button Y can input a command of firing thebullet from the shooting gun 92 b. That is, the bullet is fired in aninstant by using the manipulation input signal value of the button Ybeing changed from 0 to 1 as a trigger. The bullet is not fired in othersituations.

In the virtual space displayed on the game screen 92, the block underdropping is shot down by the bullet fired from the shooting gun 92 b.When the bullet hits the falling object, the falling object disappearsfrom the game screen 92. The mission is successful if all the blocks areshot down. The game user competes for the high score by achieving thescore according to the success or failure of the mission, the number ofthe shot-down blocks, and the smallness of the moving amount of theshooting gun 92 b in the left-and-right direction.

The drop pattern of the block in the game is determined by the gameparameters. The game parameters are generated by the converting part 42b converting the work data into the game parameters. In this applicationexample, the work data includes the painting as the content of work, thetype of paint, and a three-dimensional image including the workpiece 211and its perimeter obtained by the imaging device CA. The converting part42 b converts the surface shape of the workpiece 211 into the droppattern of the blocks under the game play by using a given conversionrule to generate the game parameters reproducing the drop pattern.

FIG. 10 is a view illustrating one example of the work data. Forexample, as illustrated in FIG. 10 , the three-dimensional image (ortwo-dimensional image) including the workpiece 211 and its perimeterobtained by the imaging device CA is converted into a planer map 91which is comprised of a given number of dots (in the example of FIG. 10, 20×30). In the map 91, “1” is assigned to the dot where the workpiece211 exists (91 a in FIGS. 10 ), and “0” is assigned to the dot where theworkpiece 211 does not exist (91 b in FIG. 10 ). Further, each line ofthe map 91 is cut out and the lines are used as line data comprised of aplurality of continuous lines (in the example of FIG. 10 , 30 lines).Then, the game parameters are generated for each line data so that theblock appears only at the dot to which “1” is assigned.

The game parameters generated by the game data generating device 42 asdescribed above are transmitted to the game terminal 3. The gamecomputer 31 of the game terminal 3 applies the game parameters to thegame program, and determines an appearing location and an appearingtiming of the block to be displayed on the game screen 92 of the display35. A filter which corrects the game parameters according to the gameuser's skill level acquired in advance may be set in the game computer31. When the game begins, a block group corresponding to the first linefirst appears at the topmost part of the screen. The appeared block(s)falls at a given rate. The falling speed of the block(s) may differaccording to the game user's skill level. When a given time passes afterthe block group corresponding to the first line appears, a block groupcorresponding to the second line appears at the topmost part of thescreen. Thus, the block group of the line corresponding to the rownumber appears at the topmost part of the screen while incrementing therow number from the first line to the last line by 1 at a constantinterval so that the appeared blocks fall one after another.

Learning Method

First, a learning method implemented by the first learning device 41 ofthe mediation server 2 is described.

A plurality of workpieces 211 are prepared for learning. The surfaceshape of the workpiece 211 for learning differs for different workpieces211. A skilled robot operator inputs a manipulation, for each of aplurality of workpieces 211 for learning, by a work sequence optimal tothe surface shape based on his/her experiences by using the robotmanipulator 213. The robot controller 201 acquires the manipulationinput signal which corresponds to an operation performed by the skilledrobot operator on the robot manipulator 213, and stores, for eachworkpiece 211, the history of the manipulation input signal so as to beassociated with the workpiece 211. For example, the history of themanipulation input signal for a certain workpiece 211 includes a numbersequence An in which the manipulation input signal value of the firstlever A is recorded at a unit-time interval, a number sequence Bn inwhich the manipulation input signal value of the second lever B isrecorded at a unit-time interval, and a number sequence Cn in which themanipulation input signal value of the button C is recorded at aunit-time interval.

The game computer 31 of the game terminal 3 stores the history of themanipulation inputted by the game user while playing the game, from thestart to the end of the game. From the history of the manipulation ofthe game user who has succeeded the mission and got the high score(hereinafter, referred to as the “high-score user”), a number sequenceXn in which the manipulation input signal value of the dial X isrecorded at a unit-time interval is created, and a number sequence Yn inwhich the manipulation input signal value of the button Y is recorded ata unit-time interval is created.

For a certain workpiece Wm, the number sequence acquired from thehistory of the manipulation of the skilled robot operator is {Anm, Bnm,Cnm}. Moreover, during the game play to which the game parametersassociated with this workpiece Wm are reflected, the number sequenceacquired from the history of the manipulation of the high-score user is{Xnm, Ynm} (m=1, 2, . . . ). Thus, one learned data set is obtained fromthe history of the manipulation of the skilled robot operator associatedwith the workpiece Wm and the history of the manipulation of thehigh-score user.

The learned data acquiring part 41 a acquires and stores a large numberof learned data sets. The pretreatment part 41 b pretreats the learneddata set and generates the teacher data set. The first learning part 41c inputs the teacher data set into the neural network, and learns arelation between the number sequence {Xn, Yn} which is the input dataand the number sequence {An, Bn, Cn} which is the output data. Variousparameters set in the neural network are stored in the parameter DB. Theneural network to which each parameter stored is set becomes the learnedmodel.

Robot Manipulating Method

Next, a robot manipulating method using the above learned model isdescribed.

FIG. 11 is a flowchart illustrating a flow of processing of a gamemethod. FIG. 11 comprehensively illustrates a series of processings fromthe preparation for the start of the game in the robot manipulatingsystem 1 until the robot 5 performs a work, where the left columnindicates processing of the game terminal 3, the center column indicatesprocessing of the mediation server 2, and the right column indicatesprocessing of the robot 5.

The game terminal 3 transmits login information along with a game startrequest to the mediation server 2 upon the start of the game (Step S01).The login information includes a game user ID etc. When the game startrequest is acquired (Step S02), the mediation server 2 transmits a workdata request to the robot 5 (Step S03). When the work data request isacquired (Step S04), the robot 5 transmits the work data to themediation server 2 in response to the acquisition (Step S05). In thisapplication example, the work data includes the content of work and acaptured image of the workpiece 211.

The mediation server 2 which acquired the work data (Step S06) generatesthe game data based on the work data (Step S07). In this applicationexample, the game data is created by converting the captured imageincluded in the work data, and the game parameters are included in thegame data. The mediation server 2 transmits the game data to the gameterminal 3 (Step S08).

The game terminal 3 which acquired the game data (Step S09) applies thegame data (game parameters) to the game program stored in advance, andexecutes the game program. At the game terminal 3, the acquired gamedata is applied to the game program, and the game is started. Themanipulation input inputted by the game user operating the gamecontroller 38 while playing the game is stored. Therefore, the gameterminal 3 starts the game to which the game parameters are reflected(Step S10). In this application example, the game user can enjoy theshooting game in which the falling blocks are shot down.

The game terminal 3 records the history of the manipulation performed bythe game user with the game controller 38 while playing the game (StepS11). When the game is ended (Step S12), the game terminal 3 transmitsthe game manipulation data including the history of the manipulationunder the game play to the mediation server 2 (Step S13). The gamemanipulation data is associated with at least one of the work data andthe game data.

In the robot control data generating device 43 of the mediation server2, the game manipulation data acquiring part 43 a acquires the gamemanipulation data (Step S14), and by the converting part 43 b convertingthe game manipulation data using the above-described learned model, itgenerates the robot control data (Step S15). The game manipulation datasent to the mediation server 2 from the game terminal 3 may include thegame score. The game score may become an index of the game user's skilllevel. When converting the game manipulation data into the robot controldata, the game user's skill level may be considered.

The mediation server 2 transmits the generated robot control data to therobot 5 (Step S16). The robot 5 acquires the robot control data (StepS17), and by operating the manipulator 204 and the end effector 206based on the robot control data, the robot 5 performs a work (Step S18).In this application example, the robot 5 performs a painting work of theworkpiece 211.

Thus, in this application example, the correlativity between the contentof game performed by the game user and the content of work performed bythe robot 5 appears to be low. The game user performs the manipulationinput to the game controller 38 while watching the virtual spaceprojected on the display 35. That is, the game user can input thecommand into the robot 5 while enjoying the game, without seeing thereal space where the robot 5 performs the work, and therefore, the robot5 can be caused to work.

As described above, the robot manipulating system 1 according to thisembodiment is characterized by including the game terminal 3 having thegame computer 31 which executes the game program, the game controller 38which accepts the input of the manipulation to the game computer 31, andthe display 35 which displays the virtual space outputted from the gamecomputer 31, the robot 5 which performs the work in the real space basedon the robot control data, and the mediation server 2 (an “informationprocessing device” in the claims) which mediates between the gameterminal 3 and the robot 5.

The mediation server 2 includes the game data feeding part 42 c whichsupplies to the game terminal 3 the game data associated with thecontent of work, the game manipulation data acquiring part 43 a whichacquires the game manipulation data including the history of the inputof the manipulation accepted by the game controller 38 while the gameprogram to which the game data is reflected is executed, the firstconverting part 43 b which converts the game manipulation data into therobot control data based on the given conversion rule, and the robotcontrol data feeding part 43 c which supplies the robot control data tothe robot 5.

Moreover, the robot manipulating method according to this embodiment isa robot manipulating method for manipulating, by using the game terminal3, the robot 5 which performs the work in the real space based on therobot control data. The method is characterized by including a gameprogram feeding step for supplying the game data associated with thecontent of work to the game terminal 3, a game manipulation dataacquiring step for acquiring the game manipulation data including thehistory of the input of the manipulation accepted by the game controller38 while the game program to which the game data is reflected isexecuted, a robot control data generating step for converting the gamemanipulation data into the robot control data based on the givenconversion rule, and a robot control data feeding step for supplying therobot control data to the robot 5.

In the robot manipulating system 1, its mediation server 2, and therobot control method, which are described above, since the gamemanipulation data is converted into the robot control data based on theconversion rule, the contents of the computing game which the game userenjoys (i.e., characters, items, effects which appear in the virtualspace of the game, the rule of the game, etc.) and the content of workperformed by the robot 5 do not need to be directly related to eachother. That is, the virtual space projected on the display 35 of thegame terminal 3 can be turned into the world unique to the game which isgreatly different from the work performed by the robot 5 in the actualworld. Therefore, the aspect of amusement of the game in which theunusual virtual space is appreciated is not spoiled.

Therefore, according to the robot manipulating system 1, its mediationserver 2, and the robot control method according to this embodiment, thegame user who is also the robot operator can make the robot 5 work bythe manipulation inputted during the game play, while purely enjoyingthe virtual world of the game.

In this embodiment, the mediation server 2 is further provided with thefirst learning part 41 c which learns the relation between the gamemanipulation data and the robot control data by using the first learneddata including the game manipulation data associated with the content ofwork and the robot control data, and generates the first learned model.Then, the first converting part 43 b inputs the game manipulation datato the first learned model which learned the relation between the gamemanipulation data and the robot control data to convert the gamemanipulation data into the robot control data.

Similarly, in this embodiment, the robot manipulating method furtherincludes a first learning step for learning the relation between thegame manipulation data and the robot control data by using the firstlearned data including the game manipulation data associated with thecontent of work and the robot control data to generate the first learnedmodel. Further, the robot control data generating step of the robotmanipulating method includes converting the game manipulation data intothe robot control data by inputting the game manipulation data to thefirst learned model which learned the relation between the gamemanipulation data and the robot control data.

Thus, by utilizing the learned model in order to convert the gamemanipulation data into the robot control data, the game manipulationdata and the robot control data can be associated with each other withhigher accuracy, even if the direct relation between the content of gameand the content of work of the robot 5 is low. That is, a degree offreedom in the content of game with respect to the content of work ofthe robot 5 can be improved.

Modification 1

Next, Modification 1 of the above embodiment is described. FIG. 12 is afunctional block diagram of a mediation server 2A of a robotmanipulating system 1 according to Modification 1. The robotmanipulating system 1 according to Modification 1 has substantially thesame configuration as the above embodiment except for a second learningdevice 44 of the mediation server 2A. Therefore, in description ofModification 1, the second learning device 44 is described in detail,and description which overlaps with the above embodiment is omitted.

The above embodiment is provided with the second converting part 42 b(game data generating part) which converts the content of work into thegame data based on the given conversion rule. In this modification, thesecond converting part 42 b also converts the work data including thecontent of work into the game data by utilizing the learned model(second learned model), similar to the first converting part 43 b (robotcontrol data generating part).

The mediation server 2A according to Modification 1 is the mediationserver 2 according to the above embodiment further provided with thesecond learning device 44. Below, description of the first learningdevice 41, the game data generating device 42, and the robot controldata generating device 43 is omitted, and the second learning device 44is described in detail.

As illustrated in FIG. 12 , the second learning device 44 includesfunctional parts of a learned data acquiring part 44 a, a pretreatmentpart 44 b, and a second learning part 44 c. The learned data acquiringpart 44 a acquires a large number of learned data sets, where eachlearned data set is game data associated with the work data. Thepretreatment part 44 b pretreats the learned data set to create theteacher data set. The second learning part 44 c inputs the teacher datasets into the neural network, and learns the relation between the workdata which is the input data and the game data which is the output data.Various parameters set in the neural network are stored in the parameterDB. The neural network to which each parameter to be stored is setbecomes the second learned model.

As described above, the mediation server 2A according to Modification 1is provided with the second learning part 44 c which learns the relationbetween the content of work and the game data using the second learneddata including the content of work and the game data associated with thecontent of work, and generates the second learned model.

Further, in the mediation server 2A, the second converting part 42 b(game data generating part) inputs the content of work to the secondlearned model which learned the relation between the content of work andthe game data to convert the content of work into the game data.

Similarly, the robot manipulating method according to Modification 1includes, in addition to the robot manipulating method according to theabove embodiment, a second learning step for learning the relationbetween the content of work and the game data using the second learneddata including the content of work and the game data associated with thecontent of work, and generating the second learned model. Further, inthe robot manipulating method according to Modification 1, the game datagenerating step of the robot manipulating method according to the aboveembodiment includes converting the content of work into the game data byinputting the content of work to the second learned model which learnedthe relation between the content of work and the game data.

Thus, by converting the content of work (work data) into the game datausing the learned model, the game data suitable for the content of workcan be created also for a more complicated content of work. Therefore,the degree of freedom in the content of game with respect to the contentof work of the robot 5 can be improved.

APPLICATION EXAMPLE 2

Below, Application Example 2 in which the robot manipulating system 1having the above configuration is applied to a robot 5A which performs aconveying work of garbage is described.

FIG. 13 is a block diagram illustrating the entire configuration of therobot manipulating system 1 according to Application Example 2. Therobot illustrated in FIG. 13 is a crane robot which agitates the garbagein a garbage pit 215. The robot 5A includes a crane 208, a gantry 209which changes the position of the crane 208 in the left-and-rightdirection and the front-and-rear direction, and a robot controller 201which controls operation of the crane 208 and the gantry 209. Thegarbage pit 215 is provided with an imaging device CA whichthree-dimensionally images the inside of the garbage pit 215.

The robot manipulator 213 has a first lever A, a second lever B, and abutton C. According to the manipulation inputted into the first lever Aand the second lever B, the gantry 209 operates and the two-dimensionalposition of the crane 208 changes. The first lever A can input a commandof the movement of the crane 208 in the front-and-rear direction and theleft-and-right direction. A manipulation input signal value inputted bythe first lever A changes continuously from 0 to 1 according to theoperating position of the first lever A. The second lever B can input acommand of the movement of the crane 208 in the up-and-down direction. Amanipulation input signal value inputted by the second lever B changescontinuously from 0 to 1 according to the operating position of thesecond lever B. The button C can input a command for gripping thegarbage by the crane 208. A manipulation input signal value inputted bythe button C changes continuously from 0 to 1 according to the operatingposition of the button C.

The garbage in the garbage pit 215 which is a work target object of therobot 5 having the above configuration is different in the surfaceheight and the property for different garbage, and the garbage in thegarbage pit 215 must be agitated so as to become more uniform. Agitatingof the garbage is performed by moving the garbage at certain coordinatesin the garbage pit 215 to other coordinates. In order to efficientlyperform the agitating work of the garbage, it is desirable that themovement locus of the crane 208 is shorter. The robot operator whooperates the robot 5A is required for high skill of determining whichcoordinates of the garbage to be selected and how the garbage to bemoved, and the order of movement.

At the game terminal 3, a game program of a puzzle game is executed.FIG. 14 illustrates one example of a game screen 92A displayed on thedisplay of the game terminal 3. Not the real space where the robot 5Aexists but a virtual space peculiar to the game is displayed on the gamescreen 92A. In the game screen 92A illustrated in FIG. 14 , a grid isdisplayed in a center part and a panel is disposed in each cell of thegrid. The panel is classified by three colors of red, blue, and purple,and if the red panel is superimposed on the blue panel, the panelchanges into purple. Other panels cannot be superimposed on the red andpurple panels. The game user can operate the game controller 38 to inputinto the game computer 31 commands related to a selection of a panel tobe moved, and a moving position of the panel.

The game controller 38 is provided with a cross key X and a button Y asthe manipulanda 381. A manipulation input signal value of the cross keyX changes continuously from 0 to 1. The cross key X can move a pointerin the left-and-right direction and the front-and-rear direction in thegame screen 92A. The manipulation input signal value of the cross key Xcorresponds to a moving direction and a moving amount of the pointer. Amanipulation input signal value of the button Y becomes a value of 0 or1 depending on whether the button Y is pushed. The button Y can acceptan input of a command of the selection. For example, the first panel isselected when the manipulation input signal value of the button Ychanges from 0 to 1 while the pointer is located on a first panel to bemoved. Moreover, when the first panel is selected and the manipulationinput signal value of the button Y changes from 0 to 1 while the pointeris located on a second panel of the moving destination, the first paneloverlaps with the second panel. The mission is successful if the colorsof all the panels become purple. The game user competes for the highscore by achieving the score according to the success or failure of themission and the smallness of the moving amount of the pointer.

The coloring pattern of the panels in the puzzle game is determined bygame parameters. The game parameters are generated by the secondconverting part 42 b converting the work data into the game parameters.In this application example, the work data includes a three-dimensionalimage inside the garbage pit 215 obtained by the imaging device CA. Theconverting part 42 b converts the surface shape of deposited garbage inthe garbage pit 215 into the coloring pattern of the panels under thegame play by using a given conversion rule to generate the gameparameters for reproducing the coloring pattern.

FIG. 15 is a view illustrating one example of the work data. Forexample, as illustrated in FIG. 15 , the three-dimensional image of thegarbage pit 215 obtained by the imaging device CA is converted into aplaner map 94 which is comprised of a given number of dots (in theexample of FIG. 15 , 15×10). The height of the deposited garbage surfaceis classified as high, medium, and low by an analyses of thethree-dimensional image, where “1” is assigned to the high dot, “0” isassigned to the medium dot, and “4” is assigned to the low dot. Further,the game parameters are generated so that the red panel appears in thedot of 1, the purple panel appears in the dot of 0, and the blue panelappears in the dot of −1. The game data includes such game parameters.

Learning Method

First, a learning method implemented by the second learning device 44 ofthe mediation server 2 is described.

A lot of learned data is prepared for learning. The learned dataincludes the surface shape of the deposited garbage in the garbage pit215, and the history of the manipulation by the skilled robot operatorwhich is associated with the surface shape. The surface shape includesthe height of the deposited garbage. The skilled robot operator inputs,using the robot manipulator 213, the manipulation in the work sequenceoptimal to the surface shape of the deposited garbage based on his/herexperiences to agitate the deposited garbage in the garbage pit 215. Therobot controller 201 acquires the manipulation input signal whichcorresponds to an operation performed by the skilled robot operator onthe robot manipulator 213, and stores the history of the manipulationinput signal so as to be associated with the surface shape of thedeposited garbage.

The three-dimensional image of the garbage pit 215 is converted into theplaner deposited garbage map 94 which is comprised of a given number ofdots (in the example of FIG. 15 , 15×10). An address is assigned to eachdot of the deposited garbage map 94, and each dot has information on thedeposited garbage height (an average value of the dot). From the historyof the manipulation input signal, the address of the dot from which thegarbage in the garbage pit 215 is taken out, and the address of the dotwhere the garbage is placed can be obtained. The dot from which thegarbage is taken out is classified as “high,” the dot where the garbageis placed is classified as “low,” and the dot with no movement of thegarbage is classified as “medium.” In a pretreatment of the learneddata, “high,” “medium,” and “low” are given to each dot of the depositedgarbage map 94 based on the foregoing rule. One set of teacher data inwhich the deposited garbage map 94 is used as the input data, and“high,” “medium,” and “low” classification of each dot of the depositedgarbage map 94 is used as the output data is created.

The learning device causes the model to learn, using a lot of teacherdata, which classification of “high,” “medium,” and “low” each dot ofthe deposited garbage map 94 falls. The game data generating device 42inputs to the learned model the deposited garbage map 94 which is thework data, obtains an output of “high,” “medium,” and “low”classification of each dot of the deposited garbage map 94, andgenerates the game data based on the output.

Robot Manipulating Method

Then, a robot manipulating method is described.

A flow of processing of the game method is substantially the same asApplication Example 1. That is, as illustrated in FIG. 11 , the gameterminal 3 transmits a game start request along with login informationto the mediation server 2 (Step S01). If the mediation server 2 acquiresthe game start request (Step S02), it transmits a work data request tothe robot 5A (Step S03). If the robot 5A acquires the work data request(Step S04), it transmits the work data to the mediation server 2 inresponse to the request (Step S05). In this application example, thework data includes the content of work and the three-dimensional imageof the garbage pit 215.

The mediation server 2 which acquired the work data (Step S06) generatesthe game data based on the work data (Step S07). In this applicationexample, the game data is created by converting the captured imageincluded in the work data into the learned model, and the game dataincludes the game parameters. The mediation server 2 transmits the gamedata to the game terminal 3 (Step S08).

The game terminal 3 which acquired the game data (Step S09) applies thegame data (game parameters) to the game program stored in advance, andexecutes the game program. At the game terminal 3, the acquired gamedata is applied to the game program, and the game is started. Themanipulation input inputted by the game user operating the gamecontroller 38 while playing the game is stored. Therefore, the gameterminal 3 starts the game to which the game parameters are reflected(Step S10). In this application example, the game user can enjoy thecolor puzzle game in which the blue and red panels are turned intopurple.

The game terminal 3 records the history of the manipulation performed bythe game user using the game controller 38 while playing the game (StepS11). If the game is ended (Step S12), the game terminal 3 transmits thegame manipulation data including the history of the manipulation underthe game play to the mediation server 2 (Step S13). The gamemanipulation data is associated with at least one of the work data andthe game data.

In the robot control data generating device 43 of the mediation server2, the game manipulation data acquiring part 43 a acquires the gamemanipulation data (Step S14), and the converting part 43 b converts thegame manipulation data into the robot control data to generate the robotcontrol data (Step S15). In this application example, the gamemanipulation data includes a combination of the coordinates of the panelto be moved and the coordinates of the destination of the movement ofthe panel, and the moving order of the panels. From such gamemanipulation data, the robot control data including the combination ofthe coordinates from which the garbage is taken out and the coordinatesinto which the garbage is carried, and the moving order of the garbagein the garbage pit 215 is generated.

The mediation server 2 transmits the generated robot control data to therobot 5A(Step S16). The robot 5A acquires the robot control data (StepS17) and operates the gantry 209 and the crane 208 based on the robotcontrol data so that the robot 5A performs the work (Step S18). In thisapplication example, the robot 5A performs the agitating work of thegarbage pit 215.

Thus, although in this application example the content of game performedby the game user and the content of work performed by the robot are notdirectly related to each other, there is a correlativity between themovement of the pointer in the game and the movement of the crane 208.However, the game user can perform the manipulation input to the gamecontroller 38 while looking at the virtual space projected on thedisplay 35, without seeing the real space where the robot 5A performsthe work so that he/she can input the command into the robot 5A whileenjoying the game to cause the robot 5A to perform the work.

Although the suitable embodiment of the present disclosure is describedabove, what changed the concrete structures and/or functions of theabove embodiment without departing from the thought of the presentdisclosure may be encompassed within the present disclosure. Theabove-described robot manipulating system 1 and robot manipulatingmethod can be change as follows.

For example, although in the above embodiment the game terminal 3, themediation server 2, and the robots 5 and 5A are independent devices, thefunction of the mediation server 2 may be provided to one of the gameterminal 3 and the robots 5 and 5A.

For example, although in the above embodiment the neural network modelperforms the learning as one example of the learning model, the learningalgorithm is not limited to this configuration. For example, othermodels different from the neural network model may perform the learningby other learning methods.

For example, in the above embodiment, the game program is stored(installed) in the game terminal 3 in advance. Although the game programis supplied to the game terminal 3 from the mediation server 2 throughthe communication apparatus 4, it may be recorded on a recording medium,such as a CD-ROM, a DVD-ROM, and a micro SD card and may be provided tothe game terminal 3. Moreover, the game program is not limited to theconfiguration in which it is stored in the game terminal 3 in advance,but the game program to which the game parameters are applied in advancemay be supplied to the game terminal 3 from the mediation server 2 eachtime he/she plays the game. In this case, the game data includes thegame program to which the game parameters are applied.

DESCRIPTION OF REFERENCE CHARACTERS

-   -   1: Robot Manipulating System    -   2, 2A: Mediation Server (One Example of Information Processing        Device)    -   3: Game Terminal    -   4: Communication Apparatus    -   5, 5A: Robot    -   20 a: Calculation Controller    -   20 b: Interface    -   21: Processor    -   22: Memory    -   24: Storage    -   25: I/O Part    -   31: Game Computer    -   35: Display    -   37: Speaker    -   38: Game Controller    -   41: First Learning Device    -   41 a: Learned Data Acquiring Part    -   41 b: Pretreatment Part    -   41 c: First Learning Part    -   42: Game Data Generating Device    -   42 a: Work Data Acquiring Part    -   42 b: Converting Part (Second Converting Part)    -   42 c: Game Data Feeding Part    -   43: Robot Control Data Generating Device    -   43 a: Game Manipulation Data Acquiring Part    -   43 b: Converting Part (First Converting Part)    -   43 c: Robot Control Data Feeding Part    -   44: Second Learning Device    -   44 a: Learned Data Acquiring Part    -   44 b: Pretreatment Part    -   44 c: Second Learning Part    -   91: Map    -   92: Game Screen    -   92 a: Line    -   92 b: Shooting Gun    -   92 c: Score    -   201: Robot Controller    -   202: Paint Feeder    -   203: Valve Device    -   204: Manipulator    -   205: Compressor    -   206: End Effector    -   206A: Painting Gun    -   207: Nozzle    -   208: Crane    -   209: Gantry    -   210: Jig    -   211: Workpiece    -   213: Robot Manipulator    -   215: Garbage Pit    -   311: Main Processor    -   312: Sound Processor    -   313: I/O Processor    -   314: Storage    -   315: Graphics Processor    -   316: Memory    -   381: Manipulandum    -   382: Manipulation Detector    -   383: Communication Device

1. An information processing device configured to mediate between a gameterminal having a game computer configured to execute a game program, agame controller configured to accept an input of manipulation into thegame computer, and a display configured to display a virtual spaceoutputted from the game computer, and a robot configured to perform awork in a real space based on robot control data, comprising: a gamedata feeding part configured to supply game data associated with thecontent of work to the game terminal; a game manipulation data acquiringpart configured to acquire game manipulation data including a history ofthe input of the manipulation accepted by the game controller while thegame program to which the game data is reflected is executed; a firstconverting part configured to convert the game manipulation data intothe robot control data based on a given conversion rule; and a robotcontrol data feeding part configured to supply the robot control data tothe robot.
 2. The information processing device of claim 1, wherein thefirst converting part converts the game manipulation data into the robotcontrol data by inputting the game manipulation data into a firstlearned model where a relation between the game manipulation data andthe robot control data is learned.
 3. The information processing deviceof claim 2, further comprising a first learning part configured to learnthe relation between the game manipulation data and the robot controldata by using first learned data including the game manipulation dataassociated with the content of work and the robot control data, andgenerate the first learned model.
 4. The information processing deviceof claim 1, further comprising a second converting part configured toconvert the content of work into the game data based on a givenconversion rule.
 5. The information processing device of claim 4,wherein the second converting part converts the content of work into thegame data by inputting the content of work into a second learned modelwhere a relation between the content of work and the game data islearned.
 6. The information processing device of claim 5, furthercomprising a second learning part configured to learn the relationbetween the content of work and the game data by using second learneddata including the content of work and the game data associated with thecontent of work, and generate the second learned model.
 7. A robotmanipulating system, comprising: a game terminal having a game computerconfigured to execute a game program, a game controller configured toaccept an input of manipulation into the game computer, and a displayconfigured to display a virtual space outputted from the game computer;a robot configured to perform a work in a real space based on robotcontrol data; and the information processing device of claim 1configured to mediate between the game terminal and the robot.
 8. Amethod of manipulating a robot configured to perform a work in a realspace based on robot control data by using a game terminal having a gamecomputer configured to execute a game program, a game controllerconfigured to accept an input of manipulation into the game computer,and a display configured to display a virtual space outputted from thegame computer, the method comprising the steps of: supplying game dataassociated with the content of work to the game terminal; acquiring gamemanipulation data including a history of the input of the manipulationaccepted by the game controller while the game program to which the gamedata is reflected is executed; converting the game manipulation datainto the robot control data based on a given conversion rule; andsupplying the robot control data to the robot.
 9. The method of claim 8,wherein the converting the game manipulation data includes convertingthe game manipulation data into the robot control data by inputting thegame manipulation data into a first learned model where a relationbetween the game manipulation data and the robot control data islearned.
 10. The method of claim 9, further comprising generating thefirst learned model by learning the relation between the gamemanipulation data and the robot control data by using first learned dataincluding the game manipulation data associated with the content of workand the robot control data.
 11. The method of claim 8, furthercomprising converting the content of work into the game data based on agiven conversion rule.
 12. The method of claim 11, wherein theconverting the content of work includes converting the content of workinto the game data by inputting the content of work into a secondlearned model where a relation between the content of work and the gamedata is learned.
 13. The method of claim 12, further comprisinggenerating the second learned model by learning the relation between thecontent of work and the game data by using second learned data includingthe content of work and the game data associated with the content ofwork.